21 research outputs found
Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
Imitation learning has traditionally been applied to learn a single task from
demonstrations thereof. The requirement of structured and isolated
demonstrations limits the scalability of imitation learning approaches as they
are difficult to apply to real-world scenarios, where robots have to be able to
execute a multitude of tasks. In this paper, we propose a multi-modal imitation
learning framework that is able to segment and imitate skills from unlabelled
and unstructured demonstrations by learning skill segmentation and imitation
learning jointly. The extensive simulation results indicate that our method can
efficiently separate the demonstrations into individual skills and learn to
imitate them using a single multi-modal policy. The video of our experiments is
available at http://sites.google.com/view/nips17intentionganComment: Paper accepted to NIPS 201
Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
In principle, reinforcement learning and policy search methods can enable
robots to learn highly complex and general skills that may allow them to
function amid the complexity and diversity of the real world. However, training
a policy that generalizes well across a wide range of real-world conditions
requires far greater quantity and diversity of experience than is practical to
collect with a single robot. Fortunately, it is possible for multiple robots to
share their experience with one another, and thereby, learn a policy
collectively. In this work, we explore distributed and asynchronous policy
learning as a means to achieve generalization and improved training times on
challenging, real-world manipulation tasks. We propose a distributed and
asynchronous version of Guided Policy Search and use it to demonstrate
collective policy learning on a vision-based door opening task using four
robots. We show that it achieves better generalization, utilization, and
training times than the single robot alternative.Comment: Submitted to the IEEE International Conference on Robotics and
Automation 201
Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning
Reinforcement learning (RL) algorithms for real-world robotic applications
need a data-efficient learning process and the ability to handle complex,
unknown dynamical systems. These requirements are handled well by model-based
and model-free RL approaches, respectively. In this work, we aim to combine the
advantages of these two types of methods in a principled manner. By focusing on
time-varying linear-Gaussian policies, we enable a model-based algorithm based
on the linear quadratic regulator (LQR) that can be integrated into the
model-free framework of path integral policy improvement (PI2). We can further
combine our method with guided policy search (GPS) to train arbitrary
parameterized policies such as deep neural networks. Our simulation and
real-world experiments demonstrate that this method can solve challenging
manipulation tasks with comparable or better performance than model-free
methods while maintaining the sample efficiency of model-based methods. A video
presenting our results is available at
https://sites.google.com/site/icml17pilqrComment: Paper accepted to the International Conference on Machine Learning
(ICML) 201
Dual Generator Offline Reinforcement Learning
In offline RL, constraining the learned policy to remain close to the data is
essential to prevent the policy from outputting out-of-distribution (OOD)
actions with erroneously overestimated values. In principle, generative
adversarial networks (GAN) can provide an elegant solution to do so, with the
discriminator directly providing a probability that quantifies distributional
shift. However, in practice, GAN-based offline RL methods have not performed as
well as alternative approaches, perhaps because the generator is trained to
both fool the discriminator and maximize return -- two objectives that can be
at odds with each other. In this paper, we show that the issue of conflicting
objectives can be resolved by training two generators: one that maximizes
return, with the other capturing the ``remainder'' of the data distribution in
the offline dataset, such that the mixture of the two is close to the behavior
policy. We show that not only does having two generators enable an effective
GAN-based offline RL method, but also approximates a support constraint, where
the policy does not need to match the entire data distribution, but only the
slice of the data that leads to high long term performance. We name our method
DASCO, for Dual-Generator Adversarial Support Constrained Offline RL. On
benchmark tasks that require learning from sub-optimal data, DASCO
significantly outperforms prior methods that enforce distribution constraint.Comment: NeurIPS 202
Time-Contrastive Networks: Self-Supervised Learning from Video
We propose a self-supervised approach for learning representations and
robotic behaviors entirely from unlabeled videos recorded from multiple
viewpoints, and study how this representation can be used in two robotic
imitation settings: imitating object interactions from videos of humans, and
imitating human poses. Imitation of human behavior requires a
viewpoint-invariant representation that captures the relationships between
end-effectors (hands or robot grippers) and the environment, object attributes,
and body pose. We train our representations using a metric learning loss, where
multiple simultaneous viewpoints of the same observation are attracted in the
embedding space, while being repelled from temporal neighbors which are often
visually similar but functionally different. In other words, the model
simultaneously learns to recognize what is common between different-looking
images, and what is different between similar-looking images. This signal
causes our model to discover attributes that do not change across viewpoint,
but do change across time, while ignoring nuisance variables such as
occlusions, motion blur, lighting and background. We demonstrate that this
representation can be used by a robot to directly mimic human poses without an
explicit correspondence, and that it can be used as a reward function within a
reinforcement learning algorithm. While representations are learned from an
unlabeled collection of task-related videos, robot behaviors such as pouring
are learned by watching a single 3rd-person demonstration by a human. Reward
functions obtained by following the human demonstrations under the learned
representation enable efficient reinforcement learning that is practical for
real-world robotic systems. Video results, open-source code and dataset are
available at https://sermanet.github.io/imitat
Learning Latent Space Dynamics for Tactile Servoing
To achieve a dexterous robotic manipulation, we need to endow our robot with
tactile feedback capability, i.e. the ability to drive action based on tactile
sensing. In this paper, we specifically address the challenge of tactile
servoing, i.e. given the current tactile sensing and a target/goal tactile
sensing --memorized from a successful task execution in the past-- what is the
action that will bring the current tactile sensing to move closer towards the
target tactile sensing at the next time step. We develop a data-driven approach
to acquire a dynamics model for tactile servoing by learning from
demonstration. Moreover, our method represents the tactile sensing information
as to lie on a surface --or a 2D manifold-- and perform a manifold learning,
making it applicable to any tactile skin geometry. We evaluate our method on a
contact point tracking task using a robot equipped with a tactile finger. A
video demonstrating our approach can be seen in https://youtu.be/0QK0-Vx7WkIComment: Accepted to be published at the International Conference on Robotics
and Automation (ICRA) 2019. The final version for publication at ICRA 2019 is
7 pages (i.e. 6 pages of technical content (including text, figures, tables,
acknowledgement, etc.) and 1 page of the Bibliography/References), while this
arXiv version is 8 pages (added Appendix and some extra details
Supervised Learning and Reinforcement Learning of Feedback Models for Reactive Behaviors: Tactile Feedback Testbed
Robots need to be able to adapt to unexpected changes in the environment such
that they can autonomously succeed in their tasks. However, hand-designing
feedback models for adaptation is tedious, if at all possible, making
data-driven methods a promising alternative. In this paper we introduce a full
framework for learning feedback models for reactive motion planning. Our
pipeline starts by segmenting demonstrations of a complete task into motion
primitives via a semi-automated segmentation algorithm. Then, given additional
demonstrations of successful adaptation behaviors, we learn initial feedback
models through learning from demonstrations. In the final phase, a
sample-efficient reinforcement learning algorithm fine-tunes these feedback
models for novel task settings through few real system interactions. We
evaluate our approach on a real anthropomorphic robot in learning a tactile
feedback task.Comment: Submitted to the International Journal of Robotics Research. Paper
length is 21 pages (including references) with 12 figures. A video overview
of the reinforcement learning experiment on the real robot can be seen at
https://www.youtube.com/watch?v=WDq1rcupVM0. arXiv admin note: text overlap
with arXiv:1710.0855
Robotic Offline RL from Internet Videos via Value-Function Pre-Training
Pre-training on Internet data has proven to be a key ingredient for broad
generalization in many modern ML systems. What would it take to enable such
capabilities in robotic reinforcement learning (RL)? Offline RL methods, which
learn from datasets of robot experience, offer one way to leverage prior data
into the robotic learning pipeline. However, these methods have a "type
mismatch" with video data (such as Ego4D), the largest prior datasets available
for robotics, since video offers observation-only experience without the action
or reward annotations needed for RL methods. In this paper, we develop a system
for leveraging large-scale human video datasets in robotic offline RL, based
entirely on learning value functions via temporal-difference learning. We show
that value learning on video datasets learns representations that are more
conducive to downstream robotic offline RL than other approaches for learning
from video data. Our system, called V-PTR, combines the benefits of
pre-training on video data with robotic offline RL approaches that train on
diverse robot data, resulting in value functions and policies for manipulation
tasks that perform better, act robustly, and generalize broadly. On several
manipulation tasks on a real WidowX robot, our framework produces policies that
greatly improve over prior methods. Our video and additional details can be
found at https://dibyaghosh.com/vptr/Comment: First three authors contributed equall